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Introduction 


Federal education policy in recent years has encouraged state and local education 
agencies to embrace data use and analysis in decision-making, ranging from policy 
development and implementation to performance evaluation. The capacity of these 
agencies to make effective and methodologically sound use of collected data for these 
purposes remains an outstanding question. As hundreds of millions of dollars are 
distributed from the federal Department of Education to states developing longitudinal 
data systems, education leaders approach data use with widely divergent levels of skill 
and understanding. 

This paper reviews four guiding principles for education stakeholders as they attempt to 
use data as a basis for making decisions. These principles draw heavily from 
experiences and observations of CNA field analysts, who are embedded in state 
departments of education to provide ongoing technical assistance. We focus on 
experiences of field analysts in Tennessee, a state currently implementing a nearly $502 
million federal Race to the Top grant — a grant largely won based on the commitment to 
data use represented by its long-standing value-added assessment system (TVAAS). As 
in states across the country, Tennessee now confronts a series of challenges in 
developing data use skills across the state education agency (SEA), local leaders, 
principals, and teachers, especially as significant portions of principal and teacher 
evaluation are now tied to student value-added data. The increasing importance of 
understanding and acting on educational data make clear the need for common guiding 
principles in their use. The principles include the following: 

• Establish common definitions and understandings of terms 

• Anticipate potential unintended consequences of data definitions and priorities 

• Ensure that data definitions are applied uniformly to appropriately homogeneous 
target populations and disparately to populations that are appropriately 
heterogeneous 

• Disaggregate data in order to reveal the most complete and accurate picture 

Based on CNA’s work in the state, we present these principles in the use of data for 
state-level decision-making. In so doing, we intend to inform a conversation about the 
appropriate role of student and teacher performance data in policy decisions and to 
prompt deeper research into how state leaders collect, analyze, and act on those sets of 
data. 
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Establish common definitions and 
understandings of terms 

Varying definitions of key data points hinder the research and policy communities in their 
efforts to understand educational needs and conditions. Especially problematic is the use 
of disparate definitions while attempting to describe the same phenomena. No Child Left 
Behind (NCLB), for example, allows states to define their own measures of student 
“proficiency” and employ their own unique achievement tests, meaning that state-by-state 
comparisons of student achievement are rendered invalid because of divergent 
definitions. Likewise, states and districts have taken differing approaches to defining high 
school dropout and graduation rates, though recent efforts by the National Governors 
Association and Council of Chief State School Officers (NGA, n.d.) have pushed states to 
adopt comparable definitions. 

Dropout Rates in Tennessee 

The What Works Clearinghouse Practice Guide on dropout prevention underscores the 
need to track data points to identify students at risk of leaving school: “Regularly 
analyzing student data is the critical first step both for determining the scope of the 
dropout problem and for identifying the specific students who are at risk of dropping out 
and should be considered for extra services or supports” (Dynarski, et al., 2008). 
However, the specific scope and the students identified will vary with the methods used 
to obtain them. 

States have generally adopted one of three approaches to reporting their dropout rates 
(REL Midwest, n.d.): 

1 ) Event Dropout Rate - the percentage of enrolled high school students who leave 
school each year without completing their high school degrees 

2) Status Dropout Rate - the percentage of 1 6-24 year-olds in a population who are 
not enrolled in school and have not earned a high school diploma or its 
equivalent. 

3) Cohort Dropout Rate - the percentage of students starting together in a specific 
grade who eventually drop out and fail to graduate. 

Dropout rate statistics from Tennessee reflect the wide variation in understanding that 
may result from using one definition of dropout rather than another, as reflected in the 
following table. 
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Dropout Rates by Cohort and Event Measures, 2007 - 2009 (TDOE, 2010). 


In Tennessee, however, the challenge of understanding the state’s dropout problem was 
further exacerbated for years by different practices among district leaders in defining the 
scope of the dropout rate. For instance, some districts did not count students requiring 
five years to complete their diplomas towards their graduation rates, while others 
included five-year graduates. 

Without clear direction from the SEA, and with little guidance from the federal Department 
of Education, districts in Tennessee took independent paths to understanding the 
prevalence of student dropout at their high schools. This patchwork approach made it 
impossible to collect consistent data across the state. 

Changes in State Report Card Standards 

Tennessee also has recently changed the standards by which it assigns “grades” to 
schools through the State Report Card. The formula for assigning grades to schools 
shifted for the 2009-10 school year, with achievement data from 2009 serving as a fixed 
transition point from the old standards and assessments to the new standards and 
assessments under the Tennessee Diploma Project (TDP) — part of the American 
Diploma Project led by Achieve and other education reform groups. 

Prior to this shift, the state set subject-specific expected growth rates based on criterion- 
reference test scores from 1998. TVAAS data tracked student progress based on 1998 
standards until new, more rigorous standards were adopted under TDP. According to the 
SEA, it “has reset the growth standard to reflect the state’s average student performance 
in 2009.” New standards, as determined by a slate of newly implemented end-of-course 
tests from grades 3 and up, will challenge schools, teachers, and students to m higher 
benchmarks that reflect “the minimal expectation for student academic progress and. ..the 
current status of educational attainment....” (TDOE, 2009). 

As a result of this change, a school rating an A in 2009 could well earn a B or C under the 
new standards for the same level of performance. Although the state department of 
education news release announcing the change in standards insisted that “These 
changes do not reflect a loss of learning but a change in the scale,...,” that change of 
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scale becomes all-important in efforts to track ongoing school performance. When such 
baseline changes in an evaluation are adopted, schools and districts have difficulty 
gauging how well they and, most important, their students, are performing over a time 
period that spans the transition. An overly hasty comparison of student and school 
performance before and after the baseline transition that relied only on school 
improvement grades instead of on the numerical school indicators behind them would 
likely be highly misleading. 

These two examples illustrate the importance of making data definitions explicit. This is 
all the more critical in view of the high stakes and significant fiscal implications now 
attached to many performance indicators. 
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Anticipate potential unintended 
consequences of data definitions and 
priorities 

By definition, unintended consequences pose perennial challenges for policy-makers and 
practitioners who, given the information available at the time of a key decision, are unable 
to foresee all of the eventual ramifications of a decision. Although many people generally 
think of unintended consequences as the unfortunate by-product of specific policies or 
actions, these consequences also result from the way data are defined and prioritized. 

Promotion and Retention Rates in Tennessee 

Recent changes in Tennessee’s approach to calculating its grade-to-grade promotion 
rate provide a good example of a decision that seems to involve a simple bookkeeping 
adjustment but, in actuality, stands to have a significant impact on students. Previously, 
the state’s promotion policies in grades K-8 were heavily influenced by the desire to 
preserve the developmental and social ties of grade-level cohorts. Thus, it was a rare 
occurrence for an eighth grade student to be held back from entering the freshman year 
in high school together with the rest of his or her peers. One result of this permissive 
approach was that a significant percentage of students entered high school with 
educational profiles indicating that they would be unlikely to graduate in four years. 
Recognizing this fact, the SEA excluded from graduation rate calculations entering high 
students whose graduation in four years, if at all, was deemed unlikely. 

Now, however, Tennessee is in the process of adopting new guidelines for calculating its 
graduation rate. These guidelines were promoted by the National Governors Association 
and endorsed by the federal government in order to achieve uniformity and comparability 
nationally in the states’ graduation rate reports (NGA, n.d.) .Once these guidelines are 
implemented, Tennessee will calculate graduation rates using the full cohort of students 
entering high school as the denominator, rather than only those whose academic profiles 
indicate a high likelihood of on-time graduation. 

This change is far from a mere bookkeeping adjustment. Students are likely to be 
affected directly by this change of definition, and in ways that may have been unforeseen 
and unintended. A possible consequence of the new guidelines, is that an increasing 
number of low-performing middle school students will now be retained at earlier grades 
for remedial work, rather than passing them along to the next grade, in the hope that they 
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will be better prepared for their transition to high school and thus more likely to graduate. 
Unfortunately, however, little research exists to validate the policy of retaining students at 
any grade. And some reviews of studies that have been undertaken indicate that 
keeping students back results in their being less, not more, likely to graduate from high 
school (Broderick, 1994). 

Unwittingly, then, in an effort to increase its graduation rates, Tennessee could end up 
doing precisely the opposite. And the lesson here may be that data definitions are far 
more powerful than we might anticipate.. They have the potential for affecting the very 
phenomena they are intended to describe. 
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Ensure data definitions are applied 
uniformly to appropriately homogeneous 
target populations and disparately to 
populations that are appropriately 
heterogeneous 

As definitions of key data points such as dropout and graduation rate evolve, educators, 
researchers, and data managers struggle to ensure accuracy in their analyses of 
collected information. When a state makes a policy decision such as Tennessee’s to 
alter its standards for assessing school performance or the formula by which graduation 
rates are calculated, the policy will likely be affected by the tension between dual needs 
for consistent enforcement and flexible response to actual conditions. This is especially 
important when these changes have potential implications for high stakes decisions - 
such as decisions about restructuring or closing poor-performing schools. And it is also 
important if our country is ever to obtain an accurate national education picture. 

In addition to the difficulties that can result from changing a data definition in mid-course 
or making comparisons between phenomena that are comparable in name only, it 
becomes problematic when states either fail to apply a data definition uniformly across a 
particular population or attempt to apply a data definition uniformly to a population within 
which there are important variations. A single example from Tennessee illustrates this 
challenge. 

Exempting middle colleges from a four-year definition of 
“graduate” in Tennessee middle colleges 

Five “middle colleges” in Tennessee (generally in the state’s more urban/ suburban areas) 
provide students the opportunity to pursue both a high school diploma and coursework 
toward the completion of a two-year post-secondary degree. The Memphis Middle 
College, hosted by Southwest Tennessee Community College, enrolls approximately 225 
students, making it the largest middle college in the state by enrollment. By comparison, 
the Nashville Middle College at Nashville State Community College enrolls only about 75 
students (TDOE State Report Cards, 2009). 

As discussed earlier in this paper, Tennessee defined the high school graduation rate for 
many years based on the number of students earning their diplomas within four years of 
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ninth grade, thus excluding students completing high school coursework in five years 
from the calculation. Under and exemption to this rule, middle colleges were granted an 
extra year to graduate students and thus maintained extremely high graduation rates 
compared to school averages statewide. In 2009, for example, both the Memphis and 
Nashville middle college graduation rates were 97 percent, according to their State 
Report Cards. 

Although all high schools in the state may now count students graduating in five years 
toward their graduation rate, the exemption previously granted the middle colleges made 
it difficult at the time to compare the effectiveness of middle colleges and traditional high 
schools. Some educators in Tennessee accused districts of sending students they 
deemed unlikely to graduate on time into middle colleges in order to buy an extra year to 
get them a diploma, thus diminishing the negative effect those students would have had 
on the districts’ overall graduation rates. On the other hand, proponents of the middle 
college exemption argued that students simultaneously pursuing courses to fulfill diploma 
requirements for high school and degree-credit courses for college deserved the chance 
to take an extra year to complete these dual tracks of study. 

Given the different missions of the middle colleges and traditional high schools, there 
seems to be at least a prima facie justification for an exemption from the four-year 
limitation for high school graduation. The real problem of comparability in this situation is 
not so much between the traditional high schools and the middle colleges as between 
high schools who could send their low-performing students to the middle colleges and 
pad their four-year graduation rates and the other high schools that had no such option. 
From an accountability standpoint, the high schools whose students have the middle 
college option have an unfair advantage over the high schools without that recourse. 
Redefining high school graduation across the board in terms of a five-year limitation has 
removed that unfairness and leveled the playing field. 
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Disaggregate data in order to reveal the 
most complete and accurate picture 

Disaggregated data permit a more fine-grained analysis than broadly categorized data. 

In certain instances, however, disaggregated data may compromise privacy (especially 
when a disaggregated population is very small) or reveal a reality that is more politically 
uncomfortable than the aggregated picture. When this is the case, officials may balk at 
releasing the disaggregated findings. Such reluctance may hinder the ability of various 
education stakeholders to understand the full complexities of the issues involved. 

With the passage of No Child Left Behind, the disaggregation of data by socio-economic 
sub-group has become a requirement for all states and has frequently given officials and 
the public a far less positive picture of education in the U.S. than the combined overview 
provides. According to Alliance for Excellent Education vice-president Phillip D.C. Lovell, 
however, several significant challenges continue to inhibit states’ compliance with the 
disaggregation requirement (Zehr, 2010): 

• Inadequate data systems 

• Unforeseen difficulty with data processing 

• Changing definitions of graduation. 

One result is that several states are still unable to report anything but a consolidated 
graduation rate. 

Disaggregation of data by both race and sex 

A recent report from the Schott Foundation found fewer than half of black male students 
nationally graduate within four years of enrolling in high school (Holzman, 201 0). The 
report notes, however, that many states, including Tennessee, publish data 
disaggregated by sex and separately by race but not by both race and sex together. (This 
limitation is also found in the periodic EDFacts reports from the U.S. Department of 
Education.) The Tennessee State Report Card lists the 2008 overall graduation rate for 
all black students at 72 percent. By comparison, the Schott report lists the 2007-08 five 
year graduation rates for black males in Tennessee at 52 percent and the on-time (fourth- 
year) rate at 44 percent, compared to 71 percent for white males for both fourth- and fifth- 
year graduates. 
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Presenting graduation rates only by racial/ethnic category and not disaggregating the 
data further by sex paints a different portrait than fully disaggregated data. In this case, it 
masks the severity of the problem of black male graduation and could possibly point to a 
different set of policy solutions than would a more nuanced picture. Although the 
aggregated view may be less alarming, lack of deeper understanding of the educational 
challenges involved can prevent the implementation of strategies that would address 
problems not fully revealed. 

Even disaggregated data, however, can do no more than provide a superficial picture that 
reflects various underlying factors and conditions that must be carefully diagnosed. Only 
that diagnosis, and not the picture itself, can provide an adequate basis for a confident 
and effective response. 
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Conclusion: Data and Action 


In a time of increasing pressure to respond to expectations for improved performance of 
students, teachers, and schools and a growing emphasis on “data-driven decision- 
making,” policy makers and education leaders understandably may be inclined to act 
quickly and decisively based on the latest data findings. Although data should indeed 
inform policy decisions, the examples cited earlier illustrate that ensuring the quality of 
those data is critically important. Rather than move immediately to develop and 
implement often costly new initiatives when current programs and policies seem 
ineffective, political and education leaders should carefully assess the strengths and 
limitations of the data at their disposal and recognize that, particularly when the data are 
unreliable, the best action may be to re-frame the issue under consideration and pursue a 
more complete analysis. 

This is precisely what officials in Tennessee did when a study commissioned by the state 
Comptroller’s Office of Research and Education Accountability cast doubt on the long- 
term effectiveness of the state’s Pre-K program in enhancing student achievement 
(Strategic Research Group, 2009). Because there were concerns about the validity of the 
study, rather than taking immediate action to alter the Pre-K program, the state continued 
to support it based on research in other states on the benefits of Pre-K, a strong basic 
belief in the value of early childhood education, and anecdotal support for the state’s 
current efforts. Then, the Tennessee Department of Education worked with Vanderbilt 
University’s Peabody Research Institute to obtain a federal Department of Education 
grant to conduct a rigorous analysis of the state’s program that will provide much more 
reliable data to facilitate a better understanding of the effects of Pre-K on Tennessee 
students and provide much better guidance for any related actions the state may want to 
take. 

The good news about the growing emphasis on data-driven decision-making is that it has 
been accompanied by a strong push - much of it driven by federal education initiatives - 
to increase the capacity of the states to collect the data required for a thorough 
assessment of state education conditions and of the impact of various efforts to improve 
them. The Tennessee experience has made clear, however, that although the increased 
focus on data in the states has great promise for facilitating important education 
improvements, this promise often fails to be realized. Even more problematically, various 
weaknesses in the data themselves, inadequate data analysis, or misplaced confidence 
in the ability of a limited set of data to indicate success or failure can seriously 
compromise a state’s efforts to move forward. 

Getting State Education Data Right: What We Can Learn from Tennessee 

11 



While the principles outlined above are grounded in CNA’s experience in Tennessee, the 
complexities of data analysis in that state will likely resonate with professionals in the field 
nationwide. Establishing a set of first principles to guide the challenging work of data 
identification, collection, analysis, and decision-making produces a framework with which 
education may find the best data to make decisions based on a comprehensive approach 
to data use. 
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